Tree-based Analysis of Simple Recurrent Network Learning

In searching for a connectionist paradigm capable of natural language processing, many researchers have explored the Simple Recurrent Network (SRN) such as Elman(1990), Cleermance(1993), Reilly(1995) and Lawrence(1996). SRNs have a context layer that keeps track of the past hidden neuron activations and enables them to deal with sequential data. The events in Natural Language span time so SRNs are needed to deal with them. Among the various levels of language processing, a phonological level can be distinguished. The Phonology deals with phonemes or graphemes the latter in the case when one works with orthographic word representations. The principles governing the combinations of these symbols is called phonotactics (Laver'1994). It is a good starting point for connectionist language analysis because there are not too many basic entities. The number of the symbols varies between 26 (for the Latin graphemes) and 50 *(for the phonemes). Recently, some experiments considering phonotactics modelling with SRNs have been carried out by Stoianov(1997), Rodd(1997). The neural network in Stoianov(1997) was trained to study the phonotactics of a large Dutch word corpus. This problem was implemented as an SRN learning task to predict the symbol following the left context given to the input layer so far. Words were applied to the network, symbol by symbol, which in turn were encoded orthogonally, that is, one node standing for one symbol (Fig.l). An extra symbol ( '#') was used as a delimiter. After the training, the network responded to the input with different neuron activations at the output layer, The more active a given output neuron is, the higher the probability is that it is a successor. The authors used a so-called optimal threshold method for establishing the threshold which determines the possible successors. This method was based on examining the network